A method lor encoding images, and an image coder 

The present invention relates to a method for encoding images 
according to the preamble of claim 1. The present invention also relates 
to a device for encoding images according to the preamble of claim 1 2. 
Furthermore, the present invention relates to an encoder according to 
the preamble of claim 23, to a decoder according to the preamble of 
claim 24, to a codec according to the preamble of claim 25, to a mobile 
terminal according to the preamble of claim 26, and a storage medium 
for storing a software program according to the preamble of claim 27. 

The image can be any digital image, a video image, a TV image, an 
image generated by a video recorder, a computer animation, a still 
image, etc. In general, a digital image consists of pixels, are arranged 
in horizontal and vertical lines, the number of which in a single image is 
typically tens of thousands. In addition, the information generated for 
each pixel contains, for instance, luminance information relating to the 
pixel, typically with a resolution of eight bits, and in colour applications 
also chrominance information, e.g. a chrominance signal. This 
chrominance signal generally consists of two components, Cb and Cr, 
which are both typically transmitted with a resolution of eight bits. On 
the basis of these luminance and chrominance values, it is possible to 
form information corresponding to the original pixel on the display 
device of a receiving video terminal. In this example, the quantity of 
data to be transmitted for each pixel is 24 bits uncompressed. Thus, the 
total amount of information for one image amounts to several megabits. 
In the transmission of a moving image, several images are transmitted 
per second. For instance in a TV image, 25 images are transmitted per 
second. Without compression, the quantity of information to be 
transmitted would amount to tens of megabits per second. However, for 
example in the Internet data network, the data transmission rate can be 
in the order of 64 kbits per second, which makes uncompressed real 
time image transmission via this network practically impossible. 

To reduce the amount of information to be transmitted, a number of 
different compression methods have been developed, such as the 
JPEG, MPEG and H.263 standards. In the transmission of video, image 
compression can be performed either as inter-frame compression, 



intra-frame . compression, or a combination of these. In inter-frame 
compression, the aim is to eliminate redundant information in 
successive image frames. Typically, images contain a large amount of 
non-varying information, for example a motionless background, or 
5 slowly changing information, for example when the subject moves 
slowly. In inter-frame compression, it is also possible to utilize motion 
compensated prediction, wherein the aim is to detect elements in the 
image which are moving, wherein motion vector and prediction error 
information are transmitted instead of transmitting the pixel values. 

10 

To enable the use of image compression techniques in real time, the 
transmitting and receiving video terminal should have a sufficiently high 
processing speed that it is possible to perform compression and 
decompression in real time. 

In several image compression techniques, an image signal in digital 
format is subjected to a discrete cosine transform (DCT) before the 
image signal is transmitted to a transmission path or stored in a storage 
means. Using a DCT, it is possible to calculate the frequency spectrum 

20 of a periodic signal, i.e. to perform a transformation from the time 
domain to the frequency domain. In this context, the word discrete 
indicates that separate pixels instead of continuous functions are 
processed in the transformation. In a digital image signal, neighbouring 
pixels typically have a substantial spatial correlation. One feature of the 

25 DCT is that the coefficients established as a result of the DCT are 
practically uncorrelated; hence, the DCT conducts the transformation of 
the image signal from the time domain to the (spatial) frequency 
domain in an efficient manner, reducing the redundancy of the image 
data. As such, use of transform coding is an effective way of reducing 

30 redundancy in both inter-frame and intra-frame coding. 

Current block-based coding methods used in still image coding and 
video coding for independently coded key frames (intra-frames) use a 
block-based approach. In general, an image is divided into NxM blocks 
35 that are coded independently using some kind of transform coding. . 
Pure block-based coding only reduces the inter-pixel correlation within 
a particular block, without considering the inter-block correlation of 
pixels. Therefore, pure block-based coding produces rather high bit 
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rates even when using transform-based coding, such as DCT coding, 
which has very efficient energy packing properties for highly correlated 
data. Therefore, current digital image coding standards exploit certain 
methods that also reduce the correlation of pixel values between 
5 blocks. 

Current digital image coding methods perform prediction in the 
transform domain, i.e. they try to predict the DCT coefficients of a block 
currently being coded using the previous coded blocks and are thus 

10 coupled with the compression method. Typically a DCT coefficient that 
corresponds to the average pixel value within an image block is 
predicted using the same DCT coefficient from the previous coded 
block. The difference between the actual and predicted coefficient is 
sent to decoder. However, this scheme can predict only the average 

1 5 pixel value, and it is not very efficient. 



C Prediction of DCT coefficients can also be performed using spatially 

5 neighbouring blocks. For example, a DCT coefficient that corresponds 

to the average pixel value within a block is predicted using the DCT 
20 coefficient(s) from a block to the left or above the current block being 
coded. DCT coefficients that correspond to horizontal frequencies (i.e. 
vertical edges) can be predicted from the block above the current block 
and coefficients that correspond to vertical frequencies (i.e. horizontal 
edges) can be predicted from the block situated to the left. Similar to 
25 the previous method, differences between the actual and predicted 
coefficients are coded and sent to the decoder. This approach allows 
prediction of horizontal and vertical edges that run through several 
blocks. 

30 In MPEG-2 compression, the DCT is performed in blocks using a block 
size of 8 x 8 pixels. The luminance level is transformed using full spatial 
resolution, while both chrominance signals are subsampled. For 
example, a field of 16 x 16 pixels is subsampled into a field of 8x8 
pixels. The differences in the block sizes are primarily due to the fact 
35 that the eye does not discern changes in chrominance equally well as 
changes in luminance, wherein a field of 2 x 2 pixels is encoded with 
the same chrominance value. 



The MPEG-2 standard defines three frame types: an l-frame (Intra), a 
P-frame (Predicted), and a B-frame (Bi-directional). An l-frame .s 
generated solely on the basis of information contained in the image 
itself wherein at the receiving end, an l-frame can be used to form the 
entire image. A P-frame is typically formed on the basis of the closest 
preceding l-frame or P-frame, wherein at the receiving stage the 
preceding l-frame or P-frame is correspondingly used together with the 
received P-frame. In the composition of P-frames, for instance motion 
compensation is used to compress the quantity of information. B- 
frames are formed on the basis of a preceding l-frame and a following 
P- or l-frame. Correspondingly, at the receiving stage it is not possible 
to compose the B-frame until the preceding and following frames have 
been received. Furthermore, at the transmission stage the order of the 
P- and B-frames is changed, wherein the P-frame following the B-frame 
is received first. This tends to accelerate reconstruction of the image in 
the receiver. 

Intra-frame coding schemes used in prior art solutions are inefficient, 
wherein transmission of intra-coded frames is bandwidth-excessive. 
This limits the usage of independently coded key frames in low bit rate 
digital image coding applications. 

The present invention addresses the problem of how to further reduce 
redundant information in image data and to produce more efficient 
coding of image data, by introducing a spatial prediction scheme 
involving the prediction of pixel values, that offers a possibility for 
prediction from several directions. This allows efficient prediction of 
edges with different orientations, resulting in considerable savings in bit 
rate. The method according to the invention also uses context- 
dependent selection of suitable prediction methods, which provides 
further savings in bit rate. 

The invention introduces a method for performing spatial prediction of 
pixel values within an image. The technical description of this document 
introduces a method and system for spatial prediction that can be used 
for block-based still image coding and for intra-frame coding in block- 
based video coders. Key elements of the invention are the use of 
multiple prediction methods and the context-dependent selection and 
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signalling of the selected prediction method. The use of multiple 
prediction methods and the context-dependent selection and signalling 
of the prediction methods allow substantial savings in bit rate to be 
achieved compared with prior art solutions. 

It is an object of the present invention to improve encoding and 
decoding of digital images such that higher encoding efficiency can be 
achieved and the bit rate of the encoded digital image can be further 
reduced. 

According to the present invention, this object is achieved by an 
encoder for performing spatially predicted encoding of image data. 



According to a first aspect of the invention there is provided a method 
15 for encoding a digital image, in which method the digital image is 
divided into blocks, characterized in that in the method a spatial 
prediction for a block is performed to reduce the amount of information 
to be transmitted, wherein at least one prediction method is defined, a 
classification is determined for at least one neighbouring block of said 
20 block to be predicted according to the contents of said neighbouring 
block, and a prediction method is selected for the current block on the 
basis of at least one said classification. 

According to a second aspect of the invention there is provided a 
25 device for encoding a digital image, which is divided into blocks, 
characterized in that the device comprises means for performing spatial 
prediction for a block to reduce the amount of information to be 
transmitted, wherein at least one prediction method has been defined, 
that the device further comprises means for determining a classification 
30 for at least one neighbouring block of said block to be predicted 
according to the contents of said neighbouring block, and means for 
selecting a prediction method for the current block on the basis of at 
least one said classification. 

35 According to a third aspect of the invention there is provided an 
encoder comprising means for encoding a digital image, and means for 
dividing the digital image into blocks, characterized in that the encoder 
comprises means for performing spatial prediction for a block to reduce 



the amount of information to be transmitted, wherein at least one 
prediction method has been defined, that the encoder further cornpnses 
means for determining a classification for at least one neighbouring 
block of said block to be predicted according to the contents of said 
neighbouring block, and means for selecting a predicts method for 
the current block on the basis of at least one said classificat.on. 

According to a fourth aspect of the invention there is provided a 
decoder comprising means for decoding a digital image, which is 
divided into blocks, characterized in that the decoder comprises means 
for performing spatial prediction for a block to reduce the amount of 
information to be transmitted, wherein at least one prediction method 
has been defined, that the decoder further comprises means for 
determining a classification for at least one neighbouring block of said 
block to be predicted according to the contents of said neighbouring 
block, and means for selecting a prediction method for the current block 
on the basis of at least one said classification. 

According to a fifth aspect of the invention there is provided a codec 
comprising means for encoding a digital image, means for dividing the 
digital image into blocks, and means for decoding a digital image, 
characterized in that the codec comprises means for performing spatial 
prediction for a block to reduce the amount of information to be 
transmitted, wherein at least one prediction method has been defined, 
that the codec further comprises means for determining a classification 
for at least one neighbouring block of said block to be predicted 
according to the contents of said neighbouring block, and means for 
selecting a prediction method for the current block on the basis of at 
least one said classification. 

According to a sixth aspect of the invention there is provided a mobile 
terminal comprising means for encoding a digital image, means for 
dividing the digital image into blocks, and means for decoding a digital 
image, characterized in that the mobile terminal comprises means for 
performing spatial prediction for a block to reduce the amount of 
information to be transmitted, wherein at least one prediction method 
has been defined, that the mobile terminal further comprises means for 
determining a classification for at least one neighbouring block of said 
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block to be predicted according to the contents of said neighbouring 
block, and means for selecting a prediction method for the current block 
on the basis of at least one said classification. 

5 According to a seventh aspect of the invention there is provided a 
storage medium for storing a software program comprising machine 
executable steps for encoding a digital image, and for dividing the 
digital image into blocks, characterized in that the software program 
further comprises machine executable steps for performing spatial 

10 prediction for a block to reduce the amount of information to be 
transmitted, wherein at least one prediction method has been defined, 
steps for determining a classification for at least one neighbouring block 
of said block to be predicted according to the contents of said 
neighbouring block, and steps for selecting a prediction method for the 

1 5 current block on the basis of at least one said classification. 

The invention is based on the idea that to perform spatial prediction of 
pixel values for a block to be coded, adjacent decoded blocks are 
examined to determine if there exists some directionality in the contents 

20 of the adjacent blocks. This directionality information is then used to 
classify the blocks. Based on the combination of the classes of the 
adjacent blocks, the contents (pixel values) of the current block are 
then predicted using a suitable prediction method. The prediction 
method is signalled to the decoder. Prediction error information is also 

25 sent if it is efficient to do that in a distortion vs. bit-rate sense. 

Considerable advantages are achieved with the present invention when 
compared with solutions of prior art. Using a method according to the 
invention, it is possible to reduce the amount of information needed 
30 when transmitting images in digital format. 

In general, the method according to the invention can be applied to 
block-based still image coding as well as to intra-frame coding in a 
block-based digital image coder. 



35 



In the following, the invention will be described in more detail with 
reference to the appended figures, in which 
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constructed 18 using one of the prediction methods in the selected 
subset and the prediction method is signalled to decoder. 

Context-dependent selection of a prediction method subset comprises 
5 directionality classification of possible neighbouring blocks, mapping of 
directionality classes to context classes and context-dependent 
selection of an appropriate prediction method subset. 

In the following, the transmission and reception of digital image frames 
10 in a transmission system is described with reference to the digital 
image transfer arrangement presented in Figure 1 . The current frame 
arrives at the transmission system 1 as input data 2 provided, for 
example, as the output of a digital video camera. The current frame 
may be provided in its entirety (i.e. a complete frame comprising NxM 
15 image blocks), in which case the frame is stored, or the transmission 
system 1 may receive the input data block by block. The blocks of the 
frame are directed one by one to a summer 4, where prediction error of 
a block is calculated e.g. by subtracting a block of the frame from a 
predicted block. The prediction error is coded in a coder 5 and decoded 
20 in a decoder 6. In summer 7 the decoded prediction error is summed 
with predicted blocks and the result is saved in a frame memory 8. The 
prediction estimator 3, where spatial prediction is performed according 
to the method of the invention, receives blocks to be used with 
prediction from the frame memory 8. 

25 

In order to form a new prediction block, the prediction estimator 3 
examines, if there exists some directionality in possible neighbouring 
blocks of the current block. This scheme is illustrated in Figure 3a. The 
reference C denotes the current block, the reference L denotes a first 

30 neighbouring block of the current block and the reference U denotes a 
second neighbouring block of the current block. In this advantageous 
embodiment of the invention, the first neighbouring block is to the left of 
the current block C and the second neighbouring block is above the 
current block C. If the scanning order is different from left to right and 

35 from top to bottom, the first neighbouring block L and the second 
neighbouring block U are not necessarily to the left of and above the 
current block C, respectively. The neighbouring blocks L, U are blocks 
adjacent to the current block C which have already been reconstructed. 
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In some embodiments of the invention more than two blocks can be 
classified and used to select the prediction method for the current block 
C. However, in the following description of a preferred embodiment of 
the invention, a maximum of two neighbouring blocks L, U are classified 
for each block C under examination. Furthermore, the classification is 
performed only if a neighbouring block L or U exists. If a current block 
does not have any neighbouring blocks, it is treated as "Non-lntra" 
during context-dependent selection of prediction methods, as will be 
explained further later in the text. 

Prediction can also be implemented in such a way that it is performed 
using only already reconstructed intra-coded blocks. In this case, all 
blocks other than intra-coded blocks are treated as "Non-lntra". 



1 5 The first neighbouring block L and the second neighbouring block U are 
H classified according to the directionality of image details inside the 

C block. As illustrated in Figure 2, directionality classifier 19 analyses the 

directionality of the neighbouring blocks using pixel value gradients. As 
a result, each neighbouring block is mapped 20 into an output class. In 
20 an advantageous embodiment of the invention there are 1 1 such output 
classes, but it is obvious that the number of output classes may vary. 
Advantageously, the output classes consist of 8 directionality classes 
DO - D7 corresponding to edge orientations k«22.5 & , k = 0, 1, .... 7 and 
3 non-directional classes D8 - D10 corresponding to flat, smooth 
25 texture and coarse texture blocks. In alternative embodiments of the 
invention, the number of directionality classes and the way in which 
they are defined may vary. 

In the system of figure 1, the prediction estimator 3 first examines if the 
30 first neighbouring block L and/or the second neighbouring block U exist. 
If either one of these blocks does not exist, that neighbouring block is 
defined as a CO block ("Non-lntra"), i.e. the current block C is on the 
edge or in a corner of the frame, or on the edge or in a corner of an 
area consisting of Intra blocks. Then, the prediction estimator 3 selects 
35 a suitable prediction method for the current block C, as described later 
in this description. Otherwise, the prediction estimator 3 calculates 
gradient information relating to the block or blocks L, U. 



There are many suitable methods for calculating the gradient 
information. In the following, one advantageous method is described. 
First, average absolute directional gradients g k) k = 0, 1, ... , 7 of a 
block L, U are defined as 
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where N is the size of the block and l(x,y) represent the pixel intensity 
values. Indices x and y refer to the co-ordinates of a pixel inside the 
10 block and k represents edge orientations. The prediction estimator 3 
calculates the gradient values g k according to the formulae above. 

Using the gradient values g k , gradient ratios r k , k = 0, 1 7 are 

defined as the ratio between the gradient value in a certain direction 
1 5 and gradient value in the orthogonal direction: 
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Based on the absolute gradient values g k and gradient ratios r k defined 
in (1) and (2), classification of the block is performed, advantageously 
according to the following classification steps 1—12 using some 
5 numerical values as thresholds. This classification process class.f.es 
3 each of the neighbouring blocks into one of a first set of block types 

^ DO— D1 0. The present invention is not limited to the values used in the 

m algorithm, but the values used in the algorithm in the following steps are 

preferred. The method can also be applied to any block size. 



In this advantageous embodiment of the invention the classification 
phase comprises 13 steps, but it is obvious that the classification may 
2 comprise also different number of steps. 
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In this step the flatness of the block is checked. Prediction estimator 3 
calculates gradient values g 0 and g 4 . These correspond to gradient 
values for horizontal (0°) and vertical (90°) image details. If both 
g 0 < 2.0 and g 4 < 2.0, the block is classified as class D8 and the initial 
20 classification process terminates. Otherwise, classification step 2 is 
performed. 

Step 2 

In this step a further check for flatness of the block is performed. The 
25 rest of the gradient values g k are calculated, and the maximum gradient 
value g max = max{g k } is determined. The maximum gradient value g max 
is compared with 2.5. If g m ax^2.5 the block is classified as class D8 
and the initial classification process terminates. Otherwise, the method 
continues from step 3. 



Step 3 

In step 3 a check for clear directionality is performed. The gradient 
ratios r k are calculated and the minimum gradient ratio r min = min{r k } is 
determined. When the minimum gradient ratio is found, the 



corresponding index k mln is defined. If r min < 0.15 the block is classified 
to corresponding class Dk min and the method continues from step 12, 
otherwise the method continues from step 4. 

5 Step 4 

In step 4 a check for texture is performed. The minimum gradient ratio 
r min is compared with 0.6. If r mln >0.6 the method continues from step 
13, otherwise the method continues from the next step. 

o 

*B 10 Step 5 

^ In step 5 the two smallest gradient ratios are checked to determine if 

S3 they are clearly distinct. The gradient ratios r h are sorted in increasing 

^ order r {0) < r {1) < r (2) ...<r (7) . Also the gradient ratio indices are 

f 3 , reordered according to the sorted order k (0 ), k (1) , k (2 ), ■-• k (7) . If 

r 15 r (1) -r OT <i(r (J) -r (1) ) the sixth classification step is performed next, 

P otherwise the method continues from the 10th classification step. 



Step 6 

In step 6 the smallest gradient ratio is checked to determine if it 
20 corresponds to directionality class D2 or D6 and the smallest gradient 
ratio is small enough. The prediction estimator 3 first examines, 
whether the index of the gradient ratio r (0 ) is either 2 or 6, wherein the 
first gradient ratio r (0) is compared with 0.6. If r m e [r k \k = 2,6} and r (0) < 

0.6, the block is classified as corresponding to class Dk<o) and the 
25 method continues from step 12. Otherwise the method continues from 
step 7. 

Step 7 

In step 7 the prediction estimator 3 first examines if the index of the 
30 second gradient ratio r (1) is either 2 or 6, wherein the first gradient ratio 
r (0) is compared with 0.6. If r m e {r t \k = 2,6) and r (0) < 0.6 the block is 

classified as corresponding to class Dk 0) and the method continues 
from the step 12, otherwise the method continues from Step 8. 

35 Step 8 

In step 8 the smallest gradient ratio is checked to determine if it 
corresponds to directionality class D1 , D3, D5 or D7 and the smallest 
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aradient ratio is small enough. The first gradient ratio r (0) is compared 
wim a5.7r ( J e {r. ,*-i,3A7> and r m < 0.5 the block is classif.ed as 
corresponding to class Dk (0) and the method continues from step 12, 
otherwise the method continues from step 9. 

Step 9 .... 
In step 9 the second gradient ratio is checked to determine if it 
corresponds to directionality class D1, D3, D5 or D7 and the smallest 
gradient ratio is small enough. The first gradient ratio r {0) is compared 
with 0.5, if r (I) e{r t | fc = 1,3,5,7}. If r (0) < 0.5 the block « classified as 
corresponding to class Dk (1) and the method continues from step 12. 
Otherwise the method continues from step 10. 

Step 1 0 

Directionality is not yet found, therefore a (somewhat) higher threshold 
value compared with the threshold value used in Step 3 can be used to 
check the directionality. This means that a more uncertain examination 
is performed. Step 10 uses the values of threshold Ti defined in Table 
1 , below. The values for Ti are compared with the first gradient ratio. If 
r ( o) < T, as defined in Table 1 , the block is classified as corresponding 
to class Dk (0) and the method continues from step 12. Otherwise the 
method continues from step 11. 



Orientation Relation for r f0 i 


Ti 


r m e[r k |* = 2,6} 


0.5 


r^efo |* = 13.5,7} 


0.4 


r m e{r k |fc = 0,4} 


0.3 



Table 1 

Step 1 1 

Directionality is not yet found, therefore in step 1 1 the three smallest 
gradient ratios are checked to determine if they are neighbours and if 
the smallest gradient ratio is in the middle. In that case a still higher 
threshold value compared with the threshold value used in Step 3 can 
be used to check the directionality. This means that a more uncertain 
examination is performed. Step 11 uses the values of threshold T 2 
defined in Table 2, below. Then, if the directionalities corresponding to 
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the second r (1) and the third gradient ratios r (2) are the closest 
neighbours for the directionality corresponding to the first gradient rat.o 
r (0) and r (0) < T 2 as defined in Table 2, the block is classified as 
corresponding to class Dk (0) and the method continues from step 12. 
Otherwise the method continues from step 1 3. 



Orientation Relation for r, m 


To 


r (0) e{r t | k = 2,6} 


0.6 


r m e{r k |* = 1,3,5,7} 


0.5 


r m e{r k |* = 0,4} 


0.4 



Table 2 



Step 12 

Step 12 performs a check that classification is really based on an edge 
in the image with a certain orientation rather than texture. Step 12 uses 
the values of threshold T 3 defined in Table 3, below. In Table 3 values 
for only two possible block sizes (8x8, 4x4) are shown, but in practical 
embodiments other block sizes can also exist, wherein respective 
values for T 3 are defined. In step 12 the minimum gradient value g min = 
min{g k } is examined. Depending on the classification and the size of the 
block, the threshold T 3 is chosen from Table 3. If g^n ^ T s the initial 
classification process terminates. Otherwise the method continues from 
step 13. 



Classification of the 
Block 


T 3 for 4x4 Block 


T 3 for 8x8 Block 


D2 and D6 


9.0 


7.0 


D1, D3, D5 and D7 


11.5 


9.0 


DO, D4 


14.0 


11.0 



Table 3 



Step 13 

Step 13 performs a check whether texture is smooth or coarse. The 
maximum gradient value g max is compared with 10.0. If g ma x^ 10-0 the 
block is classified as D9. Otherwise, the block is classified as D1 0. Step 
13 is not necessarily needed, if both smooth and coarse texture are 
mapped into the same context class. 
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Next the selection 21 of a suitable prediction method is performed for 
the current block C. In a preferred embodiment of the invention, the 
selection phase is preceded by a mapping phase. The purpose of the 
5 mapping is to reduce the memory consumption of the implementation. 
Some of the directionality classes can be mapped together. The 
classes resulting from the mapping phase are called context classes 
and they are referred to with references C1 - C6. In the preferred 
embodiment of the invention, the diagonal classes are combined to two 
10 alternative classes, one for bottom-left to top-right diagonality and the 
other for top-left to bottom-right diagonality. 

Mild and steep diagonal classes D5, D6 and D7 are mapped to the first 
diagonal context class C4. Similarly, classes D1, D2 and D3 are 
15 mapped to the second diagonal context class C2. Further, the smooth 
texture class D9 and coarse texture class D10 are mapped together to 
produce texture context class C6. This mapping is illustrated in Figure 
4. 

20 In addition to the 6 context classes C1— C6 there is one further context 
class CO used for "Non-lntra" blocks. In general, a "Non-lntra" block is a 
block that does not exist, i.e. when block C is at an image boundary. If 
the prediction is implemented in such a way that only intra-coded 
blocks are used as a reference, the definition of a "Non-lntra" block is 

25 extended to those blocks that are not intra-coded. 

In the preferred embodiment of the invention there are a total of 13 
different prediction methods, which are depicted in Figures 5a— 5p for 
8x8 blocks. Prediction methods for other block sizes and context 

30 classes can be derived in a similar fashion. In each case, prediction is 
performed in a causal manner, using neighbouring reconstructed intra- 
coded blocks L, U, UL, UR as a reference. The region used for 
prediction depends on the prediction method, as depicted in Figures 3a 
and 3b, where block C is the current block to be coded. In the case of 

35 prediction methods P1— P12, the region from which blocks may be 
used for prediction is the area covered by four neighbouring blocks L, 
UL, U and R as shown in Figure 3b. For prediction method P13, this 
region is larger, as depicted in Figure 3c. It should be appreciated that 
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in other embodiments of the invention, the number of prediction 
methods, the blocks used as prediction references, as well as the pixels 
within those blocks used to perform prediction, may vary. 

5 In an advantageous embodiment of the method according to the 
invention, a subset of prediction methods for each context class 
combination is defined and the prediction methods are prioritized 
(ranked) in each subset. Then, the prediction method used to predict 
O the content of the current block C is selected from a subset of 

43 10 prediction methods. The prediction methods within a subset differ from 
each other and correspond to those prediction methods that are most 
likely to provide an accurate prediction for block C, in the event of 
particular classifications being obtained for neighbouring blocks like L 
and U. One advantageous definition for the subsets is presented in 



hi 



15 Table 4 below. 



a 



Effectively, the results of context classification for the first neighbouring 
2 block L and second neighbouring block U are combined, i.e. both taken 

into consideration when selecting a prediction method for block C. The 
20 subset of prediction methods is selected from Table 4 according to the 
context information of the neighbouring blocks L, U. Each row of Table 
4 defines the prediction method subset for a certain pair of context 
classes for neighbouring blocks L, U and the priority (rank) of the 
prediction methods in the subset. Ranking is used to simplify the 
25 context-dependent signalling of the prediction methods, as described 
later in this description. For example, if the first neighbouring block L is 
classified into context class C2 and the second neighbouring block U is 
classified into context class C4, the subset for this combination 
comprises prediction methods P1, P9, P5, P13, P7 and P6 (in ranking 
30 order). The prediction estimator 3 further selects the most appropriate 
prediction method from this subset, as detailed later in this description. 
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In the following, the defined prediction methods are described in more 
detail. 
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Prediction method P1 

Prediction method P1 predicts the average pixel value of block C from 
the average pixel values of blocks L, UL, U and UR The average pixel 
values dL, dUL and dU of the reconstructed blocks L, UL, and U are 
calculated as the integer division defined as 



d = 



x=0,y=0 



J 



UN' 



(3) 



where N is the size of the block, l(x,y) represents the pixel intensity 
values and "//" denotes division with truncation to integer value. The 
average pixel value dC of block C is predicted according to following 
set of rules (which are written below in the form of pseudo-code): 



dC = dL 

dC = s(dL + dU-dUL) 



if all blocks L, U and UL exist, then 

if dL = dU = dUL then dC = dUL 
else if dUL = dU then dC = dL 
20 else if dUL = dL then dC = dU 

else if dL = dU then 

if chrominance prediction then 
else if | dUL - dL | < 4 then 
else dC = dL 
25 else if dUL < dL < dU then dC = dU 

else if dUL < dU < dL then dC = dL 
else if dU < dL < dUL then dC = dU 
else if dL < dU < dUL then dC = dL 
else if dL < dUL < dU OR dU < dUL < dL then 
30 dC = s(dL+dU-dUL) 

else if blocks L and U exist then dC = (dL + dU +1 ) // 2 
else if blocks L and UL exist then dC = dL 
else if blocks U and UL exist then dC = dU 
else if block L exists then dC = dL 
35 else if block U exists then dC = dU 
else if block UL exists then dC = dUL 



20 



else dC = p 

where p is a value that is in the middle of the possible pixel value 
range, e.g. 128, 7/" denotes division with truncation and s is a clipping 
5 function that restricts the values to the possible range of pixel values, 
e.g. between 0 and 255 in a system that uses an 8-bit representation of 
luminance/chrominance values. As a result, the prediction block for C is 
filled with pixels having a constant value given by dC. Prediction 
method P1 is illustrated in Figure 5a. 

10 

Prediction method P2 — P4 

Prediction methods P2 through P4 predict diagonal shapes in block C 
by extending image details from the upper right direction into block C. 
Prediction is performed by copying reference pixel values at the 
15 boundaries of blocks U and UR into block C, as depicted in Figures 5b, 
5c, 5d, respectively. Reference pixels that are marked in grey are 
connected to one or more predicted pixels. The connection is marked 
as line with dots to indicate connected predicted pixels, the value of the 
reference pixel is copied to all connected predicted pixels. 

20 

Since one or more reference blocks might be unavailable, i.e. their 
context class may be CO, prediction is performed according to following 
rules. 

25 Rule 1 

If both blocks, U and UR, are classified into one of classes C1 - C6, 
pixel prediction is performed as shown in Figures 5b, 5c and 5d 
respectively. For prediction method P2 (Figure 5b), pixels without any 
corresponding reference pixel in block UR are advantageously 
30 allocated the value of the rightmost reference pixel in block UR. 

Rule 2 

If block U is classified into one of classes C1 - C6 and block UR is 
classified as CO, pixel prediction is performed as shown in Figures 5b, 
35 5c and 5d for pixels that have a reference pixel in block U. The rest of 
the pixels are advantageously set to the value of the pixel in the lower 
right corner of the reference block U. 



Rule 3 

If block U is classified as CO, the current block C is advantageously 
filled with pixels having a constant value that is substantially in the 
middle of the possible dynamic range of pixel values, e.g. 128 (in a 
5 system, that uses an 8-bit representation of luminance/chrominance 
values). 

Prediction method P5 and P9 

Prediction methods P5 (Figure 5e) and P9 (Figure 5i) predict vertical 
1 0 and horizontal shapes in the current block C by extending image details 
into the current block C. either from above or from the left. Depending 
on the selected method (P5 or P9), the reference pixel values at the 
boundary of either block U or L are copied to the current block C as 
depicted in Figures 5e and 5i. 

15 

If the context class of the reference block is CO then the current block C 
is advantageously filled with pixels having a constant value that is 
substantially in the middle of the possible dynamic range of pixel 
values, e.g. 128 (in a system, that uses an 8-bit representation of 
20 luminance/chrominance values). 

Prediction method P6. P7 and P8 

Prediction methods P6, P7 and P8 predict diagonal shapes in the 
current block C by extending image details from the upper left direction 
25 into the current block C as depicted in Figures 5f, 5g and 5h, 
respectively. Prediction is performed by copying reference pixel values 
at the boundaries of blocks L, UL and U into the current block C 
according to following rules. 

30 Rule 1 

If all blocks L, UL and U are classified into one of classes C1 - C6, the 
pixel prediction for the current block C is performed as illustrated in 
Figures 5f, 5g and 5h. 

35 Rule 2 

If blocks UL and U are classified into one of classes C1 - C6 and block 
L is classified as CO, pixel prediction for the current block C is 
performed as shown in Figures 5f, 5g and 5h for those pixels of the 
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current block C that have a reference pixel in blocks UL and L. The 
remaining pixels in the current block C are advantageously assigned 
the value of the pixel in the lower left corner of the reference pixel area 
in block UL. 

5 

Rule 3 

If blocks L and UL are classified into one of classes C1 - C6 and block 
U is classified as CO, pixel prediction for the current block C is 
p performed as shown in Figures 5f, 5g and 5h for those pixels of the 

£ 10 current block C that have a reference pixel in blocks L and UL The 
y remaining pixels in the current block C are advantageously assigned 

m the value of the pixel in the upper right corner of the reference pixel 

Nj area in block UL. 

1" 15 Rule 4 



If blocks L and U are classified into one of classes C1 - C6 and block 
UL is classified as CO, pixel prediction for the current block C is 
performed as shown in Figures 5f, 5g and 5h for those pixels of the 
current block C that have a reference pixel in blocks L and U. Pixels 
20 with reference pixel in block UL are predicted as shown in Figures 5n, 
5o and 5p. In case of method P7, the predicted pixel value is the 
average of the two reference pixel values rounded to the nearest 
integer value, as indicated in Figure 5o. 

25 Rule 5 

If block L is classified into one of classes C1 - C6 and blocks UL and U 
are classified as CO, pixel prediction for the current block C is 
performed as shown in Figures 5f, 5g and 5h for those pixels of the 
current block C that have a reference pixel in block L. The remaining 
30 pixels in the current block C are advantageously assigned the value of 
the pixel in the upper right corner of the reference pixel area in block L. 

Rule 6 

If block UL is classified into one of classes C1 - C6 and blocks L and U 
35 are classified as CO, pixel prediction for the current block C is 
performed as shown in Figures 5f, 5g and 5h for those pixels of the 
current block C that have a reference pixel in blocks UL. Pixels of the 
current block C that have a reference pixel in block L are 




advantageously assigned the value of the lower/left reference pixel in 
block UL. Pixels of the current block C that have a reference pixel in 
block U are assigned the value of the upper/right reference pixel in 
block UL. 

5 

Rule 7 

If block U is classified into one of classes C1 - C6 and blocks L and UL 
are classified as CO, pixel prediction for the current block C is 
S performed as shown in Figures 5f, 5g and 5h for those pixels of the 
5 10 current block C that have a reference pixel in block U. The remaining 
® pixels of the current block C are advantageously assigned the value of 

J the pixel in the lower left corner of the reference pixel area in block U. 

w Rule 8 

q 15 If all blocks L, UL and L are classified as CO, the current block C is 
H advantageously filled with pixels having a constant value that is 

substantially in the middle of the possible dynamic range of pixel 
3 values, e.g. 128 (in a system, that uses an 8-bit representation of 

* luminance/chrominance values). 

20 

Prediction method P10. P11 and P12 

Prediction methods P10 through P12 predict diagonal shapes in the 
current block C by extending image details from the left into the current 
block C as depicted in Figures 5j, 5k and 51, respectively. Prediction is 
25 performed by copying reference pixel values at the boundary of blocks 
L into the current block C according to following rules. 

Rule 1 

If block L is classified into one of classes C1 - C6, the pixel prediction 
30 for the current block C is performed as illustrated in Figures 5j, 5k and 
51. Pixels of the current block C without reference pixel in block L are 
advantageously filled with the value of the pixel in the lower right corner 
of the reference pixel area. 

35 Rule 2 

If block L is classified as CO, the current block C is advantageously 
filled with pixels having a constant value that is substantially in the 



middle of the possible range of pixel values, e.g. 128 (in a system, that 
uses an 8-bit representation of luminance/chrominance values). 

Prediction method P13 
5 Prediction method P1 3 predicts the content of the current block C from 
the neighbouring image content by examining if there exists a range of 
pixels having values which substantially corresponds to the pixel values 
of the current block C. The prediction of the current block C is 
5 performed by copying reconstructed pixel values from a reference block 

S 10 B that is inside a search range SR as depicted in Figure 5m. Search 
rl range SR is defined by lists of horizontal (x) and vertical (y) 

jj displacements. Each pair of horizontal displacement and corresponding 

J vertical displacement values (x, y) defines a displacement vector 

W between the coordinates of upper left corner of the current block C and 

c 15 upper left corner of the reference block B. Prediction is allowed only for 
hi those displacements corresponding to reference block B that is 

•r completely inside the reconstructed part of the frame. Examples of 

§ displacement pairs using 512 displacements for 8x8 blocks are 

h* presented in Tables 9a and 9b. In this example the scanning order of 

20 the tables is from top-left to bottom-right row by row. In alternative 
embodiments of the invention, the search range may be different from 
that depicted in Figure 5m and / or the displacement between the 
reference block B and the current block may be defined differently. 

25 The list of allowed displacements is known to both the encoder and the 
decoder, allowing context-dependent signalling of the selected 
reference block location. 

There are many alternative ways to select the prediction method from a 
30 subset of prediction methods. For example, a cost function can be 
defined in order to evaluate the effectiveness of the different prediction 
methods of the subset to be used. The cost function may be calculated 
on the basis of information concerning the error incurred when 
predicting a current block C using a particular prediction method. This 
35 error denotes differences between actual pixel values and 
reconstructed pixel values. Typically, the error values for each pixel in 
the current block C are squared and summed together to produce a 
squared error measure for the whole block. The cost function may also 
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comprise information concerning the number of bits, i.e. the bit rate 
needed to transfer the information to the receiver. The elements of the 
cost function, particularly the bit rate, can also be weighted to 
emphasize them. One example of a cost function is: 



where cost Cx is defined as a weighted sum of distortion D and rate R 
associated with each of the prediction methods and X is the weighting 
factor. If the transmission system is band limited, the weight value is 
typically larger than if the bandwidth is wider. The values for formula (4) 
are calculated for different prediction methods and preferably that 
prediction method which yields the smallest value for the cost function 
is selected. 

Additionally, the prediction error information can also be coded prior to 
transmission to the receiver. Advantageously, there is a subset of 
coding methods defined for each prediction method. Specifically, the 
coding method could be chosen to minimise the number of bits required 
to encode the prediction error. For example, the effectiveness (bit rate) 
of the coding method is examined. 

If the prediction error is relatively small, it may not be necessary to 
transmit the prediction error information at all. 

Referring once more to Figures 1 and 2. once a suitable prediction 
method has been selected for predicting a current block C, the 
prediction estimator 3 performs spatial prediction 22 according to the 
selected prediction method. The prediction estimator 3 directs the 
reconstructed block to summer 4 where the reconstructed block is 
subtracted from the actual contents of the current block C to produce 
prediction error information for the current block. 

The encoder 1 sends 23 the information about the selected prediction 
method to the multiplexer 9, which is accompanied by displacement 
information if method P13 is used. Advantageously, the selected 
prediction method is indicated by its rank in the subset of prediction 
mehtods appropriate for the particular combination of neighbouring 



Cx = D + XR, 



(4) 




blocks (U, L) in question. Encoding of the information is advantageously 
performed using variable length coding. 

The information is further transmitted to the receiver 10, where the 
demultiplexer 1 1 demultiplexes the received information. In the receiver 
10 the prediction information is directed to the predictor 16. The 
receiver 10 also comprises a frame memory 14, where the previously 
reconstructed blocks are saved. When a new coded block arrives at the 
receiver, the predictor 1 6 performs the classifying steps for the 
neighbouring blocks U, L of the received current block C to classify 
them into directionality classes, as previously described. Then the 
predictor 16 carries out the mapping of classification information into 
context classes C1 - C6. After that the predictor 16 also examines the 
rank of the prediction method. The receiver 10 contains the information 
of the Table 4 and 5, wherein the predictor 16 can determine the 
correct prediction method according to the context class combination 
and the rank. 

When the prediction method has been determined, the predictor 16 can 
reconstruct the current block C and save it to the frame memory 14. In 
a situation where prediction error information is also received, that 
information is first decoded in the decoder 12, if necessary, and 
combined with the pixel values of the reconstructed block C. Now the 
current block C is ready to be directed to the output 15 of the receiver. 

If the prediction method of the current block C is P13, the 
reconstruction of current block C is performed in a slightly different 
manner. In this case, the receiver 10 also has to decode the 
displacement information, wherein the displacement information is used 
to copy the pixel values of the current block C from previously 
reconstructed pixel values in the frame memory 14. 

Signalling of the prediction method is advantageously based on the 
context-dependent codes defined in Table 5. After selecting the 
appropriate prediction method, the encoder 1 sends a variable length 
codeword that corresponds to the rank of the selected prediction 
method in the context-dependent subset. Advantageous examples of 
variable length codewords representing each prediction method rank 
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are listed in Table 5. For example, if the first neighbouring block L is 
classified into context class C3 and the second neighbouring block U is 
classified into context class C1, and the prediction method P9 is 
selected from the subset of the prediction methods for this combination, 
the respective rank is 2. Then, the codeword which corresponds this 
rank is "01". 



Rank 


Code 


Length 


1 


1 


1 


2 


01 


2 


3 


0000 


4 


4 


0001 


4 


5 


0010 


4 


6 


0011 


4 



Table 5 

10 The receiver 10 is aware of the contents of Table 4, i.e. it knows which 
prediction method corresponds to each of the ranks in every possible 
context (combination of classes for the neighbouring blocks L and U). 
Since the receiver 10 can derive the same context information as the 
prediction estimator 3, receiver 1 0 can associate the rank represented 

15 by the received codeword to correct prediction method and perform the 
spatial prediction for block C according to the method. 

In an advantageous embodiment of the invention the signalling of 
horizontal and vertical displacements associated with prediction method 
20 P13 is performed as follows: 
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Step 1 

Those pairs of horizontal and vertical displacements (X(i). Y(i)) that 
correspond to reference blocks B lying partially or entirely outside the 
frame are eliminated from the ordered list given in Tables 9a, 9b. The 
number of valid pairs is denoted by Nv and the ordered list of valid pairs 
which are retained after the elimination is denoted by Lv. 




Step 2 

The rank r (which is one of 1, 2, .... Nv) corresponding to the chosen 
block B within the list Lv created in Step 1 is calculated. 

5 Step 3 

Based on the value of rank r determined in Step 1 the value index, is 
calculated according to Table 6. 

Step 4 

1 0 The value index 2 = r - OffsetLow(index 1 ) is calculated using the values 
listed in Table 6. 



Range for rank r 


index. 


OffsetLow 


QffsetHIgh 


AuxLength 




(index,) 


(index,) 


(index,) 


1..-2 


1 


1 


2 


1 


3. ...,4 


2 


3 


4 


i 


5 6 


3 


5 


6 


1 


7,. ..,8 


4 


7 


8 


1 


9 12 


5 


9 


12 


2 


13 16 


6 


13 


16 


2 


17 24 


7 


17 


24 


3 


25 32 


8 


25 


32 


3 


33 48 


9 


33 


48 


4 


49, ...,64 


10 


49 


64 


4 


65.. ...96 


11 


65 


96 


5 


97. .... 128 


12 


97 


128 


5 


129, 192 


13 


129 


192 


6 


193 256 


14 


193 


256 


6 


257, .... 384 


15 


257 


384 


7 


385 512 


16 


385 


512 


7 



Table 6 



15 Step 5 

Next, a variable bits is calculated as follows. If Nv < OffsetHigh(index 1 ), 
the value for the variable bits is computed advantageously using the 
formula bits = [log2(1 + Nv - OffsetLow(indexl))], where [x] denotes the 
nearest integer > x. Otherwise, bits = AuxLength(indexi). 

20 

Step 6 

Depending on the value of Nv the variable whose sub-script is index-i is 
encoded using the corresponding Variable Length Coding given in 
Table 7 and Table 8. This codeword is transmitted to the decoder, 
25 which is illustrated with block CW1 in Figure 6. 
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f^ varia b te M is nonzero the 2^ 

encoded using a number of is i„ustrated 
bits and this codeword is transmitted to the receiver, wn 

with block CW2 in Figure 6. 



1, 


in range 
... , 16 




Af v in range 
17 32 


A/ v in range 
33, ... , 64 


VLC A 


VLC B 






VLCc 




Symbol 


Length 


Code 


Symbol 


Length 


Code 


Symbol 


Length 


Code 


A, 


2 


11 




1 


1 


c. 


2 


11 


A 2 


3 


001 




2 


01 


c 2 . 


3 


1 01 


A 3 


2 


10 


B 3 


4 


0011 


c 3 


4 


0011 


A* 


4 


0001 


B 4 


4 


0010 


C 4 


5 


00001 


As 


2 


01 


B 5 


5 


00011 


C s 


3 


100 


As 


4 


0000 


B 6 


5 


00010 


C e 


4 


0010 








B 7 


5 


00001 


C7 


4 


0001 








B a 


5 


00000 


C B 


5 


00000 














c 9 


3 


011 














C10 


3 


010 






Table 7 








A/v in range 
65, ... , 128 


Nv In range 
129, ... ,256 


Nv in range 
257, ...,512 


VLCo 


VLC E 


VLC F 




Symbol 


Length 


Code 


Symbol 


Length 


Code 


Symbol 


Length 


Code 


D, 


2 


11 




2 


11 


F, 


3 


111 


D 2 


3 


101 


E 2 


3 


101 


F 2 


4 


1011 


D 3 


5 


00001 


E 3 


4 


0111 


F 3 


4 


1010 


D A 


5 


00000 


E, 


5 


00011 


F 4 


6 


000001 


D 5 


4 


0111 


Es 


4 


0110 


F 6 


4 


1001 


D e 


4 


0110 


E 6 


5 


00010 


F e 


5. 


00001 


D 7 


3 


100 


E 7 


4 


0101 


F 7 


4 


1000 


D B 


4 


101 


E e 


4 


0100 


F a 


4 


0111 


D 9 


4 


0100 


E 9 


3 


100 


F 9 


4 


0110 


D 10 


4 


0011 


E, 0 


4 


0011 


ho 


4 


0101 


D11 


4 


0010 


En 


4 


0010 
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-1 



-7 
-19 
•15 
-7 
-8 
-1 



-8 
-13 

-6 
-_10. 
-11_ 

-8 



^8 
•15 
-7 
-9 



-8 



-14J 
-8 
-8 

~^8| 



-7 



-6 



-10 



-19 
-19 
-18 

-8_ 
-15 



j7_ 
-6 
-18 



-10 
-17 
-8 
-11 
-17 



-17_ 
-19 



-11 

-15_ 
-10 



-17 
-13 
•12 
^2 
-14 
-13 
-13 
-12 
-14 
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-14 
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-10 
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~^8| 



■13 
-10 
■10, 
-19 



-8 
■19 



-8 



-18 



-8_ 
-8 



-8 



j3 

~^8] 



-7' 
-8 
-8 
-9 



-8 



-6 



-4 



-8 
-9, 
-10 
-10 
-16 



11-16 
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-11 



-14 



•19 
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-19 

-14 

-19 

-17 

-17 

-14 

-10 

-10 

-15 

-13 

-10 



-9 



-14 
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-11 

-11 

-14 

-15 

-11 

-18 



-17 
-15 
-16 
-10 
-15 
-12_ 
-14 
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-10 

-11 

■13 

-16 

-12 

11 
-14 

-11 

-11 
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-17 
-16_ 
-11 

-9 
-17 

•19 

11 
-11 
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-14 
-15 
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-15 
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-16 
-13 
-11 
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-12 

-14 



•10 
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-2 



j3_ 
-8 
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-19 
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-5 



-12 
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-18 
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-16 
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-17 



-9, 
12. 
■13 
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-13 



-16 
^19_ 
j' 
-8 
-10 
-16 
-13_ 
-14 



-18 
-13 



-19 
-17 
-14 



-13 



-11 

-14 



■13 
-14 



-13 
-15 
-15_ 
-14 



-18 -17 



-17 -13 



-18 

^12] 

-18 

-19. 

-13 



-14 
-16 
-16 

^13| 



-14 

13 
-16 

-14_ 

-13 

-10 

-15 

-13 

-16 

-15 

-13 

-18 



Table 9b 



w 33 

^ - fl( h 0 ordered list of valid displacement 
codeword with the correct displacement vector. 

5 The blocK carrying ou, prediction ^^^j^or 
particularly advantageously ^P 1 ^^^^ processing digital 
a corresponding general purpose sing 
signals, which can be programmed »^**££m according to 
.unctions to signals received « ^ ocessor or they 

0 the invention can be which also 

can be part of the operation of such a sign* p 
contains other arrangements for signal processing. 

according to the invent.on. In an advanttyeous 

invention the software program can be read from the 

to a device comprising programmable means, e.g. a processor, 

performing the method of the invention. 
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